Efficient deduplication of randomized file paths

ABSTRACT

Disclosed are techniques for deduplicating files to be ingested by a database. A bloom filter may be built for each of a first set of files to be ingested into a data exchange to generate a set of bloom filters, wherein the data exchange includes a metadata storage where metadata including a list of files ingested is stored. The set of bloom filters may be stored in the metadata storage of the data exchange. In response to receiving a set of candidate files to be ingested into the data exchange, the set of bloom filters may be used to identify from within the set of candidate files, each candidate file that is duplicative of a file in the first set of files.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.17/709,234, filed on Mar. 30, 2022 and entitled “EFFICIENT DEDUPLICATIONOF RANDOMIZED FILE PATHS,” the disclosure of which is herebyincorporated by reference.

TECHNICAL FIELD

The present disclosure relates to data sharing platforms, andparticularly to deduplication of data being ingested into a data sharingplatform.

BACKGROUND

Databases are widely used for data storage and access in computingapplications. Databases may include one or more tables that include orreference data that can be read, modified, or deleted using queries.Databases may be used for storing and/or accessing personal informationor other sensitive information. Secure storage and access of databasedata may be provided by encrypting and/or storing data in an encryptedform to prevent unauthorized access. In some cases, data sharing may bedesirable to let other parties perform queries against a set of data.

BRIEF DESCRIPTION OF THE DRAWINGS

The described embodiments and the advantages thereof may best beunderstood by reference to the following description taken inconjunction with the accompanying drawings. These drawings in no waylimit any changes in form and detail that may be made to the describedembodiments by one skilled in the art without departing from the spiritand scope of the described embodiments.

FIG. 1A is a block diagram depicting an example computing environment inwhich the methods disclosed herein may be implemented, in accordancewith some embodiments of the present invention.

FIG. 1B is a block diagram illustrating an example virtual warehouse, inaccordance with some embodiments of the present invention.

FIG. 2 is a schematic block diagram of data that may be used toimplement a public or private data exchange, in accordance with someembodiments of the present invention.

FIG. 3 is a schematic block diagram of a deployment of a data exchange,in accordance with some embodiments of the present disclosure.

FIG. 4A is a schematic block diagram of a deployment of a data exchangethat illustrates techniques for deduplicating files that are beingingested by the data exchange, in accordance with some embodiments ofthe present invention.

FIG. 4B is a schematic block diagram of a deployment of a data exchangethat illustrates techniques for deduplicating files that are beingingested by the data exchange, in accordance with some embodiments ofthe present invention.

FIGS. 5A-5C illustrate the creation of a bloom filter for an ingestedfile and use of the bloom filter to determine if a file to be ingestedmay be duplicative of the ingested file, in accordance with someembodiments of the present invention.

FIG. 6 is a flow diagram of a method for deduplicating files to beingested by a data exchange, in accordance with some embodiments of thepresent invention.

FIG. 7 is a block diagram of an example computing device that mayperform one or more of the operations described herein, in accordancewith some embodiments of the present invention.

DETAILED DESCRIPTION

Data providers often have data assets that are cumbersome to share, butof interest to another entity. For example, a large online retailcompany may have a data set that includes the purchasing habits ofmillions of consumers over the last ten years. If the online retailerwishes to share all or a portion of this data with another entity, theonline retailer may need to use old and slow methods to transfer thedata, such as a file-transfer-protocol (FTP), or even copying the dataonto physical media and mailing the physical media to the other entity.This has several disadvantages. First, it is slow as copying terabytesor petabytes of data can take days. Second, once the data is delivered,the provider cannot control what happens to the data. The recipient canalter the data, make copies, or share it with other parties. Third, theonly entities that would be interested in accessing such a large dataset in such a manner are large corporations that can afford the complexlogistics of transferring and processing the data as well as the highprice of such a cumbersome data transfer. Thus, smaller entities (e.g.,“mom and pop” shops) or even smaller, nimbler cloud-focused startups areoften priced out of accessing this data, even though the data may bevaluable to their businesses. This may be because raw data assets aregenerally too unpolished and full of potentially sensitive data tosimply outright sell/provide to other companies. Data cleaning,de-identification, aggregation, joining, and other forms of dataenrichment need to be performed by the owner of data before it isshareable with another party. This is time-consuming and expensive.Finally, it is difficult to share data assets with many entities becausetraditional data sharing methods do not allow scalable sharing for thereasons mentioned above. Traditional sharing methods also introducelatency and delays in terms of all parties having access to the mostrecently-updated data.

Private and public data exchanges may allow data providers to moreeasily and securely share their data assets with other entities. Apublic data exchange (also referred to herein as a “Snowflake datamarketplace,” or a “data marketplace”) may provide a centralizedrepository with open access where a data provider may publish andcontrol live and read-only data sets to thousands of consumers. Aprivate data exchange (also referred to herein as a “data exchange”) maybe under the data provider's brand, and the data provider may controlwho can gain access to it. The data exchange may be for internal useonly, or may also be opened to consumers, partners, suppliers, orothers. The data provider may control what data assets are listed aswell as control who has access to which sets of data. This allows for aseamless way to discover and share data both within a data provider'sorganization and with its business partners.

The data exchange may be facilitated by a cloud computing service suchas the SNOWFLAKE™ cloud computing service, and allows data providers tooffer data assets directly from their own online domain (e.g., website)in a private online marketplace with their own branding. The dataexchange may provide a centralized, managed hub for an entity to listinternally or externally-shared data assets, inspire data collaboration,and also to maintain data governance and to audit access. With the dataexchange, data providers may be able to share data without copying itbetween companies. Data providers may invite other entities to viewtheir data listings, control which data listings appear in their privateonline marketplace, control who can access data listings and how otherscan interact with the data assets connected to the listings. This may bethought of as a “walled garden” marketplace, in which visitors to thegarden must be approved and access to certain listings may be limited.

As an example, Company A has collected and analyzed the consumptionhabits of millions of individuals in several different categories. Theirdata sets may include data in the following categories: online shopping,video streaming, electricity consumption, automobile usage, internetusage, clothing purchases, mobile application purchases, clubmemberships, and online subscription services. Company A may desire tooffer these data sets (or subsets or derived products of these datasets) to other entities, thus becoming a Data Supplier or Data Provider.For example, a new clothing brand may wish to access data sets relatedto consumer clothing purchases and online shopping habits. Company A maysupport a page on its website that is or functions substantially similarto a data exchange, where a data consumer (e.g., the new clothing brand)may browse, explore, discover, access and potentially purchase data setsdirectly from Company A. Further, Company A may control: who can enterthe data exchange, the entities that may view a particular listing, theactions that an entity may take with respect to a listing (e.g., viewonly), and any other suitable action. In addition, a data provider maycombine its own data with other data sets from, e.g., a public dataexchange (also referred to as a “data marketplace”), and create newlistings using the combined data.

A data exchange may be an appropriate place to discover, assemble,clean, and enrich data to make it more monetizable. A large company on adata exchange may assemble data from across its divisions anddepartments, which could become valuable to another company. Inaddition, participants in a private ecosystem data exchange may worktogether to join their datasets together to jointly create a useful dataproduct that any one of them alone would not be able to produce. Oncethese joined datasets are created, they may be listed on the dataexchange or on the data marketplace.

Sharing data may be performed when a data provider creates a shareobject (hereinafter referred to as a share) of a database in the dataprovider's account and grants the share access to particular objects(e.g., tables, secure views, and secure user-defined functions (UDFs))of the database. Then, a read-only database may be created usinginformation provided in the share. Access to this database may becontrolled by the data provider. A “share” encapsulates all of theinformation required to share data in a database. A share may include atleast three pieces of information: (1) privileges that grant access tothe database(s) and the schema containing the objects to share, (2) theprivileges that grant access to the specific objects (e.g., tables,secure views, and secure UDFs), and (3) the consumer accounts with whichthe database and its objects are shared. The consumer accounts withwhich the database and its objects are shared may be indicated by a listof references to those consumer accounts contained within the shareobject. Only those consumer accounts that are specifically listed in theshare object may be allowed to look up, access, and/or import from thisshare object. By modifying the list of references of other consumeraccounts, the share object can be made accessible to more accounts or berestricted to fewer accounts.

In some embodiments, each share object contains a single role. Grantsbetween this role and objects define what objects are being shared andwith what privileges these objects are shared. The role and grants maybe similar to any other role and grant system in the implementation ofrole-based access control. By modifying the set of grants attached tothe role in a share object, more objects may be shared (by adding grantsto the role), fewer objects may be shared (by revoking grants from therole), or objects may be shared with different privileges (by changingthe type of grant, for example to allow write access to a shared tableobject that was previously read-only). In some embodiments, shareobjects in a provider account may be imported into the target consumeraccount using alias objects and cross-account role grants.

When data is shared, no data is copied or transferred between users.Sharing is accomplished through the cloud computing services of a cloudcomputing service provider such as SNOWFLAKE™. Shared data may then beused to process SQL queries, possibly including joins, aggregations, orother analysis. In some instances, a data provider may define a sharesuch that “secure joins” are permitted to be performed with respect tothe shared data. A secure join may be performed such that analysis maybe performed with respect to shared data but the actual shared data isnot accessible by the data consumer (e.g., recipient of the share).

A data exchange may also implement role-based access control to governaccess to objects within consumer accounts using account level roles andgrants. In one embodiment, account level roles are special objects in aconsumer account that are assigned to users. Grants between theseaccount level roles and database objects define what privileges theaccount level role has on these objects. For example, a role that has ausage grant on a database can “see” this database when executing thecommand “show databases”; a role that has a select grant on a table canread from this table but not write to the table. The role would need tohave a modify grant on the table to be able to write to it.

One way a customer may ingest data into a data exchange is to uploadfiles into e.g., a cloud storage (such as Amazon S3™) and then importthe files into the data exchange. In order to prevent ingestion ofduplicate copies of files (i.e., files that are already within the dataexchange) and wasting storage space, the data exchange may performdeduplication on files that a user is ingesting into the data exchange.Indeed, many data exchanges provide guarantees on the amount of time thedata of an ingested file will not be duplicated in a subsequentlyingested file (e.g., 65 days). When ingesting new files, if the dataexchange determines that any of the new files contain data that isduplicative of data within files that have already been ingested by thedata exchange, it may skip ingestion of that file and may only ingestfiles that are determined to be new (i.e., not duplicative). There are anumber of techniques that data exchanges may utilize in order todeduplicate files. For example, a data exchange may maintain metadatastores where metadata including a list of files ingested is stored. Thismetadata may include file path information that is used to perform e.g.,file path-based minimum/maximum pruning. Such file path information mayinclude various information that may be used to prune files to beingested. For example, the file path for a file may includeyear/month/date information indicating a date when the file was ingestedand the year/month/date information of newly received files may be used(e.g., compared to year/month/date information from the file paths ofalready ingested files) to prune newly received files based on whetherthey are outside or inside specific date ranges etc. as indicated bytheir file paths.

However, most customers do not have files with well-structured filepaths. Instead, customers often have files with unclustered file paths(e.g., ‘s3://bucket-name/<Random-UUID>.csv’) that are essentially randomstrings, so that the use of minimum/maximum pruning will result in alarge number of false matches due to the random distribution ofinformation in the file paths. As a result, the data exchange could endup scanning the entire file load history. Caching generally does nothelp in these scenarios as the data exchange would have to cache theentire file load history to efficiently deduplicate new files. Thus,deduplication is often inefficient for customers who have unclusteredfile paths.

The present disclosure addresses the above and other issues by providingtechniques for deduplicating files to be ingested by a data exchange. Inresponse to receiving a first set of files to be ingested into the dataexchange, a processing device may build a bloom filter for each of thefirst set of files. The set of bloom filters may be stored in adedicated slice of a metadata storage of the data exchange. The metadatastorage may also include file loading metadata of the first set of filesin a separate dedicated slice. In response to subsequently receiving aset of candidate files to be ingested into the data exchange, theprocessing device may utilize the file loading metadata of the first setof files and the file loading metadata of the set of candidate files toperform standard deduplication techniques (e.g., file path-basedminimum/maximum deduplication) and thereby remove one or more of the setof candidate files that are duplicative of a file in the first set offiles. This may result in a reduced set of candidate files. Theprocessing device may retrieve and cache each bloom filter from themetadata storage and for each of the reduced set of candidate files,process the candidate file with the bloom filter for each of the firstset of files to determine if the candidate file is duplicative of any ofthe first set of files. Candidate files that are not duplicative areidentified and set for ingestion, while candidate files that arepotentially duplicative are identified and set for further scanning.

FIG. 1A is a block diagram of an example computing environment 100 inwhich the systems and methods disclosed herein may be implemented. Inparticular, a cloud computing platform 110 may be implemented, such asAmazon Web Services™ (AWS), Microsoft Azure™, Google Cloud™, or thelike. As known in the art, a cloud computing platform 110 providescomputing resources and storage resources that may be acquired(purchased) or leased and configured to execute applications and storedata.

The cloud computing platform 110 may host a cloud computing service 112that facilitates storage of data on the cloud computing platform 110(e.g. data management and access) and analysis functions (e.g. SQLqueries, analysis), as well as other computation capabilities (e.g.,secure data sharing between users of the cloud computing platform 110).The cloud computing platform 110 may include a three-tier architecture:data storage 140, query processing 130, and cloud services 120.

Data storage 140 may facilitate the storing of data on the cloudcomputing platform 110 in one or more cloud databases 141. Data storage140 may use a storage service such as Amazon S3™ to store data and queryresults on the cloud computing platform 110. In particular embodiments,to load data into the cloud computing platform 110, data tables may behorizontally partitioned into large, immutable files which may beanalogous to blocks or pages in a traditional database system. Withineach file, the values of each attribute or column are grouped togetherand compressed using a scheme sometimes referred to as hybrid columnar.Each table has a header which, among other metadata, contains theoffsets of each column within the file.

In addition to storing table data, data storage 140 facilitates thestorage of temp data generated by query operations (e.g., joins), aswell as the data contained in large query results. This may allow thesystem to compute large queries without out-of-memory or out-of-diskerrors. Storing query results this way may simplify query processing asit removes the need for server-side cursors found in traditionaldatabase systems.

Query processing 130 may handle query execution within elastic clustersof virtual machines, referred to herein as virtual warehouses or datawarehouses. Thus, query processing 130 may include one or more virtualwarehouses 131, which may also be referred to herein as data warehouses.The virtual warehouses 131 may be one or more virtual machines operatingon the cloud computing platform 110. The virtual warehouses 131 may becompute resources that may be created, destroyed, or resized at anypoint, on demand. This functionality may create an “elastic” virtualwarehouse that expands, contracts, or shuts down according to the user'sneeds. Expanding a virtual warehouse involves generating one or morecompute nodes 132 to a virtual warehouse 131. Contracting a virtualwarehouse involves removing one or more compute nodes 132 from a virtualwarehouse 131. More compute nodes 132 may lead to faster compute times.For example, a data load which takes fifteen hours on a system with fournodes might take only two hours with thirty-two nodes.

Cloud services 120 may be a collection of services that coordinateactivities across the cloud computing service 112. These services tietogether all of the different components of the cloud computing service112 in order to process user requests, from login to query dispatch.Cloud services 120 may operate on compute instances provisioned by thecloud computing service 112 from the cloud computing platform 110. Cloudservices 120 may include a collection of services that manage virtualwarehouses, queries, transactions, data exchanges, and the metadataassociated with such services, such as database schemas, access controlinformation, encryption keys, and usage statistics. Cloud services 120may include, but not be limited to, authentication engine 121,infrastructure manager 122, optimizer 123, exchange manager 124,security engine 125, and metadata storage 126.

FIG. 1B is a block diagram illustrating an example virtual warehouse131. The exchange manager 124 may facilitate the sharing of data betweendata providers and data consumers, using, for example, a data exchange.For example, cloud computing service 112 may manage the storage andaccess of a database 108. The database 108 may include various instancesof user data 150 for different users, e.g. different enterprises orindividuals. The user data 150 may include a user database 152 of datastored and accessed by that user. The user database 152 may be subjectto access controls such that only the owner of the data is allowed tochange and access the user database 152 upon authenticating with thecloud computing service 112. For example, data may be encrypted suchthat it can only be decrypted using decryption information possessed bythe owner of the data. Using the exchange manager 124, specific datafrom a user database 152 that is subject to these access controls may beshared with other users in a controlled manner. In particular, a usermay specify shares 154 that may be shared in a public or data exchangein an uncontrolled manner or shared with specific other users in acontrolled manner as described above. A “share” encapsulates all of theinformation required to share data in a database. A share may include atleast three pieces of information: (1) privileges that grant access tothe database(s) and the schema containing the objects to share, (2) theprivileges that grant access to the specific objects (e.g., tables,secure views, and secure UDFs), and (3) the consumer accounts with whichthe database and its objects are shared. When data is shared, no data iscopied or transferred between users. Sharing is accomplished through thecloud services 120 of cloud computing service 112.

Sharing data may be performed when a data provider creates a share of adatabase in the data provider's account and grants access to particularobjects (e.g., tables, secure views, and secure user-defined functions(UDFs)). Then a read-only database may be created using informationprovided in the share. Access to this database may be controlled by thedata provider.

Shared data may then be used to process SQL queries, possibly includingjoins, aggregations, or other analysis. In some instances, a dataprovider may define a share such that “secure joins” are permitted to beperformed with respect to the shared data. A secure join may beperformed such that analysis may be performed with respect to shareddata but the actual shared data is not accessible by the data consumer(e.g., recipient of the share). A secure join may be performed asdescribed in U.S. application Ser. No. 16/368,339, filed Mar. 18, 2019.

User devices 101-104, such as laptop computers, desktop computers,mobile phones, tablet computers, cloud-hosted computers, cloud-hostedserverless processes, or other computing processes or devices may beused to access the virtual warehouse 131 or cloud service 120 by way ofa network 105, such as the Internet or a private network.

In the description below, actions are ascribed to users, particularlyconsumers and providers. Such actions shall be understood to beperformed with respect to devices 101-104 operated by such users. Forexample, notification to a user may be understood to be a notificationtransmitted to devices 101-104, an input or instruction from a user maybe understood to be received by way of the user's devices 101-104, andinteraction with an interface by a user shall be understood to beinteraction with the interface on the user's devices 101-104. Inaddition, database operations (joining, aggregating, analysis, etc.)ascribed to a user (consumer or provider) shall be understood to includeperforming of such actions by the cloud computing service 112 inresponse to an instruction from that user.

FIG. 2 is a schematic block diagram of data that may be used toimplement a public or data exchange in accordance with an embodiment ofthe present invention. The exchange manager 124 may operate with respectto some or all of the illustrated exchange data 200, which may be storedon the platform executing the exchange manager 124 (e.g., the cloudcomputing platform 110) or at some other location. The exchange data 200may include a plurality of listings 202 describing data that is sharedby a first user (“the provider”). The listings 202 may be listings in adata exchange or in a data marketplace. The access controls, management,and governance of the listings may be similar for both a datamarketplace and a data exchange.

The listing 202 may include access controls 206, which may beconfigurable to any suitable access configuration. For example, accesscontrols 206 may indicate that the shared data is available to anymember of the private exchange without restriction (an “any share” asused elsewhere herein). The access controls 206 may specify a class ofusers (members of a particular group or organization) that are allowedto access the data and/or see the listing. The access controls 206 mayspecify that a “point-to-point” share in which users may request accessbut are only allowed access upon approval of the provider. The accesscontrols 206 may specify a set of user identifiers of users that areexcluded from being able to access the data referenced by the listing202.

Note that some listings 202 may be discoverable by users without furtherauthentication or access permissions whereas actual accesses are onlypermitted after a subsequent authentication step (see discussion ofFIGS. 4 and 6 ). The access controls 206 may specify that a listing 202is only discoverable by specific users or classes of users.

Note also that a default function for listings 202 is that the datareferenced by the share is not exportable by the consumer.Alternatively, the access controls 206 may specify that this is notpermitted. For example, access controls 206 may specify that secureoperations (secure joins and secure functions as discussed below) may beperformed with respect to the shared data such that viewing andexporting of the shared data is not permitted.

In some embodiments, once a user is authenticated with respect to alisting 202, a reference to that user (e.g., user identifier of theuser's account with the virtual warehouse 131) is added to the accesscontrols 206 such that the user will subsequently be able to access thedata referenced by the listing 202 without further authentication.

The listing 202 may define one or more filters 208. For example, thefilters 208 may define specific identity data 214 (also referred toherein as user identifiers) of users that may view references to thelisting 202 when browsing the catalog 220. The filters 208 may define aclass of users (users of a certain profession, users associated with aparticular company or organization, users within a particulargeographical area or country) that may view references to the listing202 when browsing the catalog 220. In this manner, a private exchangemay be implemented by the exchange manager 124 using the samecomponents. In some embodiments, an excluded user that is excluded fromaccessing a listing 202, i.e. adding the listing 202 to the consumedshares 156 of the excluded user, may still be permitted to view arepresentation of the listing when browsing the catalog 220 and mayfurther be permitted to request access to the listing 202 as discussedbelow. Requests to access a listing by such excluded users and otherusers may be listed in an interface presented to the provider of thelisting 202. The provider of the listing 202 may then view demand foraccess to the listing and choose to expand the filters 208 to permitaccess to excluded users or classes of excluded users (e.g., users inexcluded geographic regions or countries).

Filters 208 may further define what data may be viewed by a user. Inparticular, filters 208 may indicate that a user that selects a listing202 to add to the consumed shares 156 of the user is permitted to accessthe data referenced by the listing but only a filtered version that onlyincludes data associated with the identifier 214 of that user,associated with that user's organization, or specific to some otherclassification of the user. In some embodiments, a private exchange isby invitation: users invited by a provider to view listings 202 of aprivate exchange are enabled to do by the exchange manager 124 uponcommunicating acceptance of an invitation received from the provider.

In some embodiments, a listing 202 may be addressed to a single user.Accordingly, a reference to the listing 202 may be added to a set of“pending shares” that is viewable by the user. The listing 202 may thenbe added to a group of shares of the user upon the user communicatingapproval to the exchange manager 124.

The listing 202 may further include usage data 210. For example, thecloud computing service 112 may implement a credit system in whichcredits are purchased by a user and are consumed each time a user runs aquery, stores data, or uses other services implemented by the cloudcomputing service 112. Accordingly, usage data 210 may record an amountof credits consumed by accessing the shared data. Usage data 210 mayinclude other data such as a number of queries, a number of aggregationsof each type of a plurality of types performed against the shared data,or other usage statistics. In some embodiments, usage data for a listing202 or multiple listings 202 of a user is provided to the user in theform of a shared database, i.e. a reference to a database including theusage data is added by the exchange manager 124 to the consumed shares156 of the user.

The listing 202 may also include a heat map 211, which may represent thegeographical locations in which users have clicked on that particularlisting. The cloud computing service 112 may use the heat map to makereplication decisions or other decisions with the listing. For example,a data exchange may display a listing that contains weather data forGeorgia, USA. The heat map 211 may indicate that many users inCalifornia are selecting the listing to learn more about the weather inGeorgia. In view of this information, the cloud computing service 112may replicate the listing and make it available in a database whoseservers are physically located in the western United States, so thatconsumers in California may have access to the data. In someembodiments, an entity may store its data on servers located in thewestern United States. A particular listing may be very popular toconsumers. The cloud computing service 112 may replicate that data andstore it in servers located in the eastern United States, so thatconsumers in the Midwest and on the East Coast may also have access tothat data.

The listing 202 may also include one or more tags 213. The tags 213 mayfacilitate simpler sharing of data contained in one or more listings. Asan example, a large company may have a human resources (HR) listingcontaining HR data for its internal employees on a data exchange. The HRdata may contain ten types of HR data (e.g., employee number, selectedhealth insurance, current retirement plan, job title, etc.). The HRlisting may be accessible to 100 people in the company (e.g., everyonein the HR department). Management of the HR department may wish to addan eleventh type of HR data (e.g., an employee stock option plan).Instead of manually adding this to the HR listing and granting each ofthe 100 people access to this new data, management may simply apply anHR tag to the new data set and that can be used to categorize the dataas HR data, list it along with the HR listing, and grant access to the100 people to view the new data set.

The listing 202 may also include version metadata 215. Version metadata215 may provide a way to track how the datasets are changed. This mayassist in ensuring that the data that is being viewed by one entity isnot changed prematurely. For example, if a company has an original dataset and then releases an updated version of that data set, the updatescould interfere with another user's processing of that data set, becausethe update could have different formatting, new columns, and otherchanges that may be incompatible with the current processing mechanismof the recipient user. To remedy this, the cloud computing service 112may track version updates using version metadata 215. The cloudcomputing service 112 may ensure that each data consumer accesses thesame version of the data until they accept an updated version that willnot interfere with current processing of the data set.

The exchange data 200 may further include user records 212. The userrecord 212 may include data identifying the user associated with theuser record 212, e.g. an identifier (e.g., warehouse identifier) of auser having user data 151 in service database 158 and managed by thevirtual warehouse 131.

The user record 212 may list shares associated with the user, e.g.,reference listings 154 created by the user. The user record 212 may listshares consumed by the user, e.g. reference listings 202 created byanother user and that have been associated to the account of the useraccording to the methods described herein. For example, a listing 202may have an identifier that will be used to reference it in the sharesor consumed shares 156 of a user record 212.

The listing 202 may also include metadata 204 describing the shareddata. The metadata 204 may include some or all of the followinginformation: an identifier of the provider of the shared data, a URLassociated with the provider, a name of the share, a name of tables, acategory to which the shared data belongs, an update frequency of theshared data, a catalog of the tables, a number of columns and a numberof rows in each table, as well as name for the columns. The metadata 204may also include examples to aid a user in using the data. Such examplesmay include sample tables that include a sample of rows and columns ofan example table, example queries that may be run against the tables,example views of an example table, example visualizations (e.g., graphs,dashboards) based on a table's data. Other information included in themetadata 204 may be metadata for use by business intelligence tools,text description of data contained in the table, keywords associatedwith the table to facilitate searching, a link (e.g., URL) todocumentation related to the shared data, and a refresh intervalindicating how frequently the shared data is updated along with the datethe data was last updated.

The metadata 204 may further include category information indicating atype of the data/service (e.g., location, weather), industry informationindicating who uses the data/service (e.g., retail, life sciences), anduse case information that indicates how the data/service is used (e.g.,supply chain optimization, or risk analysis). For instance, retailconsumers may use weather data for supply chain optimization. A use casemay refer to a problem that a consumer is solving (i.e., an objective ofthe consumer) such as supply chain optimization. A use case may bespecific to a particular industry, or can apply to multiple industries.Any given data listing (i.e., dataset) can help solve one or more usecases, and hence may be applicable to multiple use cases.

Because use case information relates to how data is used, it can be apowerful tool for organizing/searching for data listings as it allowsconsumers of the data marketplace to explore and find datasets andservices based on industry problems they're trying to solve (e.g.,supply chain optimization, audience segmentation). However, providersoften describe use cases for data listings in an unstructured format,making it hard for consumers to find them. Because there is nostandardized representation for such use case information, it isdifficult to create data listing filters based on use case information.

Embodiments of the present disclosure solve the above and other problemsby enabling providers to assign use case data to data listings in astructured manner, thereby allowing for data listings to be organizedand searched/filtered based on use case information in a more effectivemanner. A processing device may be used to assign to a first datalisting, a set of use cases from a plurality of use cases, each of theset of use cases indicating a manner in which data of the first datalisting is used. In order to perform this assigning, the processingdevice may provide a listing creation interface having selectableindications of each of the plurality of use cases and may receive, viathe listing creation interface, a selection of the set of use casesassigned to the first data listing (e.g., from the consumer). The firstdata listing may be published on the data exchange, wherein the firstdata listing is one of a plurality of data listings published on thedata exchange and the processing device may provide a data listinginterface comprising a graphical representation of each of the pluralityof data listings; and an interactable menu including a selectableindication of each of the plurality of use cases. In response toreceiving a selection of one or more of the plurality of use cases viathe interactable menu, the processing device may display in the datalisting interface, a graphical representation of each of the pluralityof data listings that have been assigned any of the selected one or moreuse cases. The embodiments described herein make it easy for consumersto browse the data exchange based on their business needs in order tofind listings that solve those needs. Embodiments of the presentdisclosure also enable a data exchange operator to learn aboutconsumers' business needs based on their browsing patterns and queryingactivities (individual and collective), and further personalize theiroverall data exchange experience (listing recommendations on worksheets,etc.). It should be noted that a “business need” and a “use case” areused interchangeably herein.

The exchange data 200 may further include a catalog 220. The catalog 220may include a listing of all available listings 202 and may include anindex of data from the metadata 204 to facilitate browsing and searchingaccording to the methods described herein. In some embodiments, listings202 are stored in the catalog in the form of JavaScript Object Notation(JSON) objects.

Note that where there are multiple instances of the virtual warehouse131 on different cloud computing platforms, the catalog 220 of oneinstance of the virtual warehouse 131 may store listings or referencesto listings from other instances on one or more other cloud computingplatforms 110. Accordingly, each listing 202 may be globally unique(e.g., be assigned a globally unique identifier across all of theinstances of the virtual warehouse 131). For example, the instances ofthe virtual warehouses 131 may synchronize their copies of the catalog220 such that each copy indicates the listings 202 available from allinstances of the virtual warehouse 131. In some instances, a provider ofa listing 202 may specify that it is to be available on only specifiedone or more computing platforms 110.

In some embodiments, the catalog 220 is made available on the Internetsuch that it is searchable by a search engine such as the Bing™ searchengine or the Google search engine. The catalog may be subject to asearch engine optimization (SEO) algorithm to promote its visibility.Potential consumers may therefore browse the catalog 220 from any webbrowser. The exchange manager 124 may expose uniform resource locators(URLs) linked to each listing 202. This URL may be searchable and can beshared outside of any interface implemented by the exchange manager 124.For example, the provider of a listing 202 may publish the URLs for itslistings 202 in order to promote usage of its listing 202 and its brand.

FIG. 3 illustrates a cloud environment 300 which includes a storageplatform 310 (similar to the cloud computing platform 110 illustrated inFIG. 1A) and a cloud deployment 305. The cloud deployment 305 maycomprise a similar architecture to cloud computing service 112(illustrated in FIG. 1A) and may be a deployment of a data exchange ordata marketplace. Although illustrated with a single cloud deployment,the cloud environment 300 may have multiple cloud deployments which maybe physically located in separate remote geographical regions but mayall be deployments of a single data exchange or data marketplace.Although embodiments of the present disclosure are described withrespect to a data exchange, this is for example purpose only and theembodiments of the present disclosure may be implemented in anyappropriate enterprise database system or data sharing platform wheredata may be shared among users of the system/platform.

The cloud deployment 305 may include hardware such as processing device305A (e.g., processors, central processing units (CPUs), memory 305B(e.g., random access memory (RAM), storage devices (e.g., hard-diskdrive (HDD), solid-state drive (SSD), etc.), and other hardware devices(e.g., sound card, video card, etc.). A storage device may comprise apersistent storage that is capable of storing data. A persistent storagemay be a local storage unit or a remote storage unit. Persistent storagemay be a magnetic storage unit, optical storage unit, solid statestorage unit, electronic storage units (main memory), or similar storageunit. Persistent storage may also be a monolithic/single device or adistributed set of devices. The cloud deployment 305 may comprise anysuitable type of computing device or machine that has a programmableprocessor including, for example, server computers, desktop computers,laptop computers, tablet computers, smartphones, set-top boxes, etc. Insome examples, the cloud deployment 305 may comprise a single machine ormay include multiple interconnected machines (e.g., multiple serversconfigured in a cluster).

Databases and schemas may be used to organize data stored in the clouddeployment 305 and each database may belong to a single account withinthe cloud deployment 305. Each database may be thought of as a containerhaving a classic folder hierarchy within it. Each database may be alogical grouping of schemas and a schema may be a logical grouping ofdatabase objects (tables, views, etc.). Each schema may belong to asingle database. Together, a database and a schema may comprise anamespace. When performing any operations on objects within a database,the namespace is inferred from the current database and the schema thatis in use for the session. If a database and schema are not in use forthe session, the namespace must be explicitly specified when performingany operations on the objects.

The storage platform 310 may facilitate the storing of data and maycomprise any appropriate object storage service such as e.g., the AmazonS3™ service to store data and query results. The storage platform 310may comprise multiple buckets (databases) 311A-311C.

FIG. 3 also illustrates an example data ingestion process via whichfiles are added to a stage and then loaded into cloud deployment 305. Inthe example of FIG. 3 , cloud deployment 305 may include an externalstage 320 which may specify an external location (e.g., S3 bucket) wheredata files to be ingested are stored so that the data in the files canbe loaded into table 325. The external stage 320 may be an object thatincludes information pertaining to the storage of the files to beingested, including the bucket (e.g., bucket 311A) where the files arestored, the named storage integration object or credentials for thebucket (if it is protected), and an encryption key (if the files in thebucket have been encrypted). A customer may reference (e.g., as part ofan expression to copy data into table 325) the external stage 320 as thelocation (file path prefix) where the data is stored. For example,customers may write pipelines to copy data from external stage 320 intotable 325 of the cloud deployment 305. In this way, the external stage320 may allow customers to load tables from and export tables to buckets311A-311C on storage platform 310. Although FIG. 3 illustrates anexample using an external stage, any appropriate means of importing datamay be used. For example, a customer may directly ingest files from thebuckets 311A-311C, or the cloud deployment 305 may enable continuousloading of files as soon as the files available in a stage, rather thanmanually executing statements/expressions (e.g., a copy statement) on aschedule to load larger batches.

Because customers may ingest files in bursts (e.g., small batches offiles at a time), the cloud deployment 305 may accumulate the smallbatches that the customer has been ingesting and periodically compactall of the small batches accumulated during a predefined time periodinto a combined file (this is referred to herein as the compaction phaseof file ingestion). In this way, the cloud deployment 305 may obtain thelist of files in the combined file using a single API call. Thepredefined time period during which the cloud deployment 305 willaccumulate small batches of files before compacting them may be referredto as the compaction interval and may be any appropriate time period(e.g., a day, a week). The cloud deployment 305 may perform thiscompaction at the end of each compaction interval.

When the combined file is ingested, the cloud deployment 305 maygenerate file loading metadata including a list of files ingested. Thefile loading metadata may be used to prevent reloading the same files(and duplicating data) in a table. The file loading metadata maycomprise the path (i.e. prefix) and name of each loaded file (e.g.,‘s3://bucket-name/<Random-UUID>.csv’), and may be used by the clouddeployment 305 for deduplication purposes including preventing loadingof files with the same name even if they were later modified (e.g., havea different eTag) and performing file path-based minimum/maximum pruning(also referred to herein as minimum and maximum pruning). The fileloading metadata may be stored in metadata store 315, which may be anyappropriate metadata storage e.g., Foundation database (FDB). Themetadata store 315 may include a number of data persistence objects(DPOs—not shown) in which data pertaining to the data exchange may bestored. For example, a base dictionary DPO (not shown) may comprise aset of database tables used to store information about the definition ofa database of the cloud deployment 305 including information aboutdatabase objects such as tables, indexes, columns, datatypes, and views.The metadata store 315 may also include an EP file DPO (not shown) inwhich the file loading metadata may be stored. More specifically, thefile loading metadata may be stored in slice 315A of the EP file DPO.

FIG. 4A illustrates the cloud deployment 305 ingesting a first set offiles, in accordance with some embodiments of the present disclosure.The memory 305B may include a data deduplication module 306, which maybe executed by processing device 305A in order to perform the functionsdescribed herein. It should be noted that although embodiments of thepresent disclosure are described with respect to data being ingestedinto a data exchange, this is for example purposes only and is not alimitation. The embodiments of the present disclosure may be utilizedfor deduplicating data that is being ingested into any appropriatestorage and/or analysis platform such as cloud storage, a storage device(e.g., hard-disk drive (HDD), solid-state drive (SSD), etc.), or anyother appropriate platform.

As discussed herein, the processing device 305A may continuously intakesmall batches of files. When the current compaction interval ends, theprocessing device 305A may initiate compaction of all of the smallbatches of files accumulated during the compaction interval (alsoreferred to as the first set of files). During this compaction phase,the processing device 305A may generate and store the file loadingmetadata and use the file loading metadata to perform standarddeduplication tasks such as minimum/maximum pruning (as discussedhereinabove) to generate a reduced set of files. For ease of descriptionand illustration it will be assumed that all of the files in the firstset of files are new (not duplicative) and are fully ingested.

Upon ingesting the first set of files, the processing device 305A maybuild a bloom filter for each of the files in the first set of files.FIGS. 5A and 5B illustrate the process of generating a bloom filter 500defined for a file P (e.g., one of the first set of files). The bloomfilter 500 may be a bit array of m bits, all of which are initially setto 0 as shown in FIG. 5A. There may also be k different hash functionsdefined as part of the bloom filter 500 (shown as X, Y, and Zrespectively in FIG. 5A), each of which maps or hashes the correspondingfile (file P in the example of FIGS. 5A and 5B) to one of the m arraypositions (as discussed in further detail herein), thereby generating auniform random distribution. Typically, k is a small constant whichdepends on the desired false positive rate (FPR), while m isproportional to k and the number of elements to be added. The processingdevice 305A may set k for each bloom filter based on desired performancemetrics including false positivity rate (FPR) and processing time. Forexample, a bloom filter created with a larger number of hash functionsmay provide a lower FPR but may require longer processing time.Conversely, a bloom filter created with a smaller number of hashfunctions may provide a higher FPR but may require less processing time.As shown in FIG. 5A, the bloom filter 500 is defined with 15 bits, and 3hash functions X, Y, and Z.

It should be noted that in scenarios where a current small batch offiles is in the process of being ingested (i.e., has not been fullyingested) at the end of the compaction interval, the processing device305A may refrain from creating bloom filters for the files of thecurrent small batch of files, and instead may accumulate the currentsmall batch of files along with small batches of files ingested during asubsequent compaction interval.

Referring now to FIG. 5B, the bloom filter 500 may hash the file P witheach of the hash functions X, Y, and Z, such that each of the hashfunctions X, Y, and Z may generate an output between e.g., 0-1000. Theprocessing device 305A may mark the position of the array correspondingto the output of each of the hash functions as 1. In the example of FIG.5B, hash function X generates an output of 5 (thus the 5^(th) bit of thearray is marked as 1), hash function Y generates an output of 9 (thusthe 9^(th) bit of the array is marked as 1), and hash function Zgenerates an output of 11 (thus the 11^(th) bit of the array is markedas 1).

Referring back to FIG. 4A, the bloom filters generated for each of thefirst set of files may be collectively referred to (and shown in FIG.4A) as bloom filter data. The processing device 305A may store the bloomfilter data in a dedicated slice 315B of the metadata store 315. It ismore advantageous from a computing and storage resources perspective tosave the bloom filter data instead of recomputing it every time a smallbatch of files needs to be deduplicated. In addition, it is undesirableto store the bloom filter data along with the file loading metadata inslice 315A because a transition from previous deduplication models tothe bloom filter-based deduplication model described herein should beseamless. In addition, it is desirable to be able to roll back to aprevious deduplication model if an error occurs. Indeed, currentdeduplication techniques operate by fetching file metadata which hasproven to be time consuming. Storing the bloom filter data separately ina metadata store allows for more performant reads and for downloading ofthe file metadata to be skipped if the bloom filter determines thatduplicates are not present. Storing the bloom filter data along withfile metadata may negate these benefits. Thus, the processing device305A may add a new slice 315B to the metadata store 315 (within the EPfile DPO). The processing device 305A may then serialize and store thebloom filter data in the slice 315B. The slice 315B may have the samekeys as slice 315A while the values will correspond to the serializedbloom filter for each of the files in the first set of files. Forexample, the keys of the slice 315B may include e.g., ACCOUNT_ID,TABLE_ID, FILE_VERSION, ID etc., while the values of the slice 315B mayinclude the bloom filter for each of the files in the first set offiles.

The processing device 305A may read the bloom filter data more quicklythan it would file loading metadata, and the volume of data is smallerand may be more efficiently cached. In some examples, a cache sizesimilar to the cache size used to cache file loading metadata may holdbloom filter data for 20 times the amount of files. This reduces thememory and network overhead during deduplication and also allows alarger number of files to be cached. In one example, assuming there areapproximately 22 million files being ingested and that a bloom filter isto be generated for each one of them, and taking the average size of thebloom filter to be e.g., 2400 bytes, the size of slice 315B will beapproximately 53 GB. Because the logic of the deduplication module 306is run on multiple virtual warehouse instances, this 53 GB can beefficiently cached across all of those virtual warehouse instances. Itshould be noted that approximately 53 GB of bloom filter datacorresponds to multiple terabytes of file loading metadata, which isessentially uncacheable.

Referring now to FIG. 4B, a new compaction interval may begin and theprocessing device 305A may begin to ingest batches of files (referred toas candidate files). At the end of the compaction interval, theprocessing device 305A (executing module 306) may begin the compactionphase. During the compaction phase, the processing device 305A maygenerate and store the file loading metadata and commence the standardminimum/maximum pruning operation to generate a reduced set of candidatefiles.

During deduplication and after the minimum/maximum pruning operation,for each of the reduced set of candidate files, the processing device305A may retrieve the bloom filter for each of the first set of filesfrom the slice 315B of the metadata store 315, and query each of thebloom filters to see if the data of the candidate file is duplicative ofthe data within any of the first set of files. More specifically, theprocessing device 305A may retrieve (lookup) from the metadata store315, deserialize, and cache the bloom filter generated for each of thefirst set of files. In some embodiments, the processing device 305A mayuse point lookups when retrieving the bloom filters for the first set offiles from the metadata store 315. Indeed, the metadata store 315 mayprovide optimal performance when it is queried for values thatcorrespond to a specified key that is of interest, (i.e., a pointlookup). The performance of the metadata store 315 may be limited if theprocessing device 305A simply requests all values from a certainrange/set of keys (i.e., a scan).

The processing device 305A may cache the retrieved bloom filters in adedicated cache 309. For example, a cache of size 0.5 GB should be ableto hold bloom filters for approximately 800 million files.

Because the processing device 305A may perform point lookups for each ofthe reduced set of files, for unclustered file paths, existingminimum/maximum-based pruning strategies require the processing device305A to look up all of the files that belong to the table 325, whereasfor well clustered file paths, this number should be close to zero. Thisis because when the customer's file paths are randomly distributed(unclustered) then existing minimum/maximum-based pruning strategies areinefficient and will result in a search of all the files ingested by thetable 325.

Referring to FIG. 5C, when the processing device 305A queries the bloomfilter 500 to see if a particular candidate file Q is duplicative of thefile P that the bloom filter 500 corresponds to (i.e., contains datathat is duplicative of the data within the file P), the bloom filter 500may perform the 3 hash functions X, Y, and Z on the candidate file Q. Ifthe output of all 3 hash operations each correspond to a bit of thearray that has been marked with a 1, the processing device 305A maydetermine that the data of the candidate file Q is potentially withinthe file P. If the output of any of the hash functions X, Y, and Zcorresponds to a bit of the array that has not been marked with a 1, theprocessing device 305A may determine that the data of the candidate fileQ is definitely is not duplicative of the data in the file P. As shownin the example of FIG. 5C, the hash function Y has generated an outputthat corresponds to the 8^(th) bit of the array, marked with 0. Thus,the processing device 305A may determine that the candidate file Q isnot duplicative of the file P, and proceed to compare the candidate fileQ to the bloom filter of the next file among the first set of files.

In this way, for each candidate file of the reduced set of candidatefiles (the files to be deduplicated), the processing device 305A mayprocess that candidate file with the bloom filter of each of the firstset of files, and determine whether the candidate file is located withinany of the first set of files (i.e., contains data that is duplicativeof the data within any of the first set of files). Each candidate filethat is new (i.e., does not contain data that is duplicative of the datain any of the first set of files) may be grouped together by theprocessing device 305A to create a further reduced set of candidatefiles that is to be loaded and completely scanned (i.e., ingested). Theprocessing device 305A may also build and store a bloom filter for eachof the further reduced set of candidate files.

In some embodiments, the processing device 305A may prune each candidatefile that is determined to potentially have duplicative data. In otherembodiments, for each of the candidate files that the processing device305A determines is potentially duplicative, the processing device 305Amay perform a second check by downloading the history of ingested filesfrom the storage platform 310 and determining if the candidate fileexists in the history of ingested files. If the processing device 305Adetermines that the candidate file is not within the history of ingestedfiles, it may determine that the candidate file needs to be ingested. Ifthe candidate file is located within the history of ingested files, itmay determine that the candidate file is a duplicate and prune it.

FIG. 6 is a flow diagram of a method 600 for deduplicating, inaccordance with some embodiments of the present disclosure. Method 600may be performed by processing logic that may comprise hardware (e.g.,circuitry, dedicated logic, programmable logic, a processor, aprocessing device, a central processing unit (CPU), a system-on-chip(SoC), etc.), software (e.g., instructions running/executing on aprocessing device), firmware (e.g., microcode), or a combinationthereof. In some embodiments, the method 600 may be performed byprocessing device 305A of cloud deployment 305 (illustrated in FIGS. 4Aand 4B).

Referring simultaneously to FIGS. 4A and 4B, the deployment 305 maycontinuously intake small batches of files. When the current compactioninterval ends, the processing device 305A may initiate compaction of allof the small batches of files accumulated during the compaction interval(also referred to as a first set of files). During this compactionphase, the processing device 305A may generate and store the fileloading metadata and use the file loading metadata to perform thestandard deduplication tasks such as minimum/maximum pruning (asdiscussed hereinabove) to generate a reduced set of files. For ease ofdescription and illustration it will be assumed that all of the files inthe first set of files are new (not duplicative) and are fully ingested.At block 605, upon ingesting the first set of files, the processingdevice 305A may build a bloom filter for each of the files in the firstset of files.

The bloom filters generated for each of the first set of files may becollectively referred to (and shown in FIG. 4A) as bloom filter data. Atblock 610, the processing device 305A may store the bloom filter data ina dedicated slice 315B of the EP file DPO. It is more advantageous froma computing and storage resources perspective to save the bloom filterdata instead of recomputing it every time a small batch of files needsto be deduplicated. In addition, it is undesirable to store the bloomfilter data along with the file loading metadata in slice 315A becausewhen a transition occurs the transition should be seamless. In addition,it is desirable to be able to roll back if something goes wrong. Thus,the processing device 305A may add a new slice 315B to the EP file DPO.The processing device 305A may then serialize and store the bloom filterdata in the slice 315B. The slice 315B may have the same keys as slice315A while the values will correspond to the serialized bloom filter foreach of the files in the first set of files. For example, the keys ofthe slice 315B may include e.g., ACCOUNT_ID, TABLE_ID, EP_FILE_VERSION,ID etc., while the values of the slice 315B may include the bloom filterdata for each of the files in the first set of files.

A new compaction interval may begin and the processing device 305A maybegin to ingest small batches of files. At the end of the compactioninterval, the processing device 305A (executing module 306) may beginthe compaction phase. During the compaction phase, the processing device305A may generate and store the file loading metadata and at block 615,may commence the standard minimum/maximum pruning operation to generatea reduced set of candidate files.

During deduplication and after the minimum/maximum pruning operation,for each of the reduced set of files (referred to as candidate files),the processing device 305A may retrieve the bloom filter for each of thefirst set of files from the slice 315B of the metadata store 315, and atblock 620 may query each of the bloom filters to see if the data of thecandidate file is duplicative of the data within any of the first set offiles. More specifically, the processing device 305A may retrieve(lookup) from the metadata store 315, deserialize, and cache the bloomfilter generated for each of the first set of files. The processingdevice 305A may cache the retrieved bloom filters in a dedicated cache309. For example, a cache of size 0.5 GB should be able to hold bloomfilters for ˜800 million files. In some embodiments, the processingdevice 305A may use point lookups when retrieving the bloom filters forthe first set of files from the metadata store 315. Indeed, the metadatastore 315 may provide optimal performance when it is queried for valuesthat correspond to a specified key that is of interest, (i.e., a pointlookup). The performance of the metadata store 315 may be limited if theprocessing device 305A simply requests all values from a certainrange/set of keys (i.e., a scan).

In this way, for each candidate file of the reduced set of files (thefiles to be deduplicated), the processing device 305A may run the bloomfilter of each of the first set of files against that candidate file,and determine whether the candidate file is located within any of thefirst set of files (i.e., contains data that is duplicative of the datawithin any of the first set of files). Each candidate file that is new(i.e., does not contain data that is duplicative of the data in any ofthe first set of files) may be grouped together by the processing device305A to create a further reduced set of files that is to be downloadedand completely scanned (i.e., completely ingested). The processingdevice 305A may also build and store a bloom filter for each of thescanned and loaded files of the further reduced set of files. Theprocessing device 305A may prune each candidate file that is determinedto potentially have duplicative data.

In some embodiments, for each of the candidate files that the processingdevice 305A determines is potentially duplicative, the processing device305A may download the history from cloud provider storage and perform asecond check to determine if the candidate file needs to be ingested.

FIG. 7 illustrates a diagrammatic representation of a machine in theexample form of a computer system 700 within which a set ofinstructions, for causing the machine to perform any one or more of themethodologies discussed herein for deduplicating files to be ingested bya data exchange.

In alternative embodiments, the machine may be connected (e.g.,networked) to other machines in a local area network (LAN), an intranet,an extranet, or the Internet. The machine may operate in the capacity ofa server or a client machine in a client-server network environment, oras a peer machine in a peer-to-peer (or distributed) networkenvironment. The machine may be a personal computer (PC), a tablet PC, aset-top box (STB), a Personal Digital Assistant (PDA), a cellulartelephone, a web appliance, a server, a network router, a switch orbridge, a hub, an access point, a network access control device, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein. In one embodiment,computer system 700 may be representative of a server.

The exemplary computer system 700 includes a processing device 702, amain memory 704 (e.g., read-only memory (ROM), flash memory, dynamicrandom access memory (DRAM), a static memory 705 (e.g., flash memory,static random access memory (SRAM), etc.), and a data storage device718, which communicate with each other via a bus 730. Any of the signalsprovided over various buses described herein may be time multiplexedwith other signals and provided over one or more common buses.Additionally, the interconnection between circuit components or blocksmay be shown as buses or as single signal lines. Each of the buses mayalternatively be one or more single signal lines and each of the singlesignal lines may alternatively be buses.

Computing device 700 may further include a network interface device 707which may communicate with a network 720. The computing device 700 alsomay include a video display unit 710 (e.g., a liquid crystal display(LCD) or a cathode ray tube (CRT)), an alpha-numeric input device 712(e.g., a keyboard), a cursor control device 714 (e.g., a mouse) and anacoustic signal generation device 715 (e.g., a speaker). In oneembodiment, video display unit 710, alphanumeric input device 712, andcursor control device 714 may be combined into a single component ordevice (e.g., an LCD touch screen).

Processing device 702 represents one or more general-purpose processingdevices such as a microprocessor, central processing unit, or the like.More particularly, the processing device may be complex instruction setcomputing (CISC) microprocessor, reduced instruction set computer (RISC)microprocessor, very long instruction word (VLIW) microprocessor, orprocessor implementing other instruction sets, or processorsimplementing a combination of instruction sets. Processing device 702may also be one or more special-purpose processing devices such as anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), a digital signal processor (DSP), network processor,or the like. The processing device 702 is configured to execute datadeduplication instructions 725, for performing the operations and stepsdiscussed herein.

The data storage device 718 may include a machine-readable storagemedium 728, on which is stored one or more sets of data deduplicationinstructions 725 (e.g., software) embodying any one or more of themethodologies of functions described herein. The data deduplicationinstructions 725 may also reside, completely or at least partially,within the main memory 704 or within the processing device 702 duringexecution thereof by the computer system 700; the main memory 704 andthe processing device 702 also constituting machine-readable storagemedia. The data deduplication instructions 725 may further betransmitted or received over a network 720 via the network interfacedevice 707.

The machine-readable storage medium 728 may also be used to storeinstructions to perform the methods described herein. While themachine-readable storage medium 728 is shown in an exemplary embodimentto be a single medium, the term “machine-readable storage medium” shouldbe taken to include a single medium or multiple media (e.g., acentralized or distributed database, or associated caches and servers)that store the one or more sets of instructions. A machine-readablemedium includes any mechanism for storing information in a form (e.g.,software, processing application) readable by a machine (e.g., acomputer). The machine-readable medium may include, but is not limitedto, magnetic storage medium (e.g., floppy diskette); optical storagemedium (e.g., CD-ROM); magneto-optical storage medium; read-only memory(ROM); random-access memory (RAM); erasable programmable memory (e.g.,EPROM and EEPROM); flash memory; or another type of medium suitable forstoring electronic instructions.

Unless specifically stated otherwise, terms such as “receiving,”“routing,” “granting,” “determining,” “publishing,” “providing,”“designating,” “encoding,” or the like, refer to actions and processesperformed or implemented by computing devices that manipulates andtransforms data represented as physical (electronic) quantities withinthe computing device's registers and memories into other data similarlyrepresented as physical quantities within the computing device memoriesor registers or other such information storage, transmission or displaydevices. Also, the terms “first,” “second,” “third,” “fourth,” etc., asused herein are meant as labels to distinguish among different elementsand may not necessarily have an ordinal meaning according to theirnumerical designation.

Examples described herein also relate to an apparatus for performing theoperations described herein. This apparatus may be specially constructedfor the required purposes, or it may comprise a general purposecomputing device selectively programmed by a computer program stored inthe computing device. Such a computer program may be stored in acomputer-readable non-transitory storage medium.

The methods and illustrative examples described herein are notinherently related to any particular computer or other apparatus.Various general purpose systems may be used in accordance with theteachings described herein, or it may prove convenient to construct morespecialized apparatus to perform the required method steps. The requiredstructure for a variety of these systems will appear as set forth in thedescription above.

The above description is intended to be illustrative, and notrestrictive. Although the present disclosure has been described withreferences to specific illustrative examples, it will be recognized thatthe present disclosure is not limited to the examples described. Thescope of the disclosure should be determined with reference to thefollowing claims, along with the full scope of equivalents to which theclaims are entitled.

As used herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”,“comprising”, “includes”, and/or “including”, when used herein, specifythe presence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. Therefore, the terminology usedherein is for the purpose of describing particular embodiments only andis not intended to be limiting.

It should also be noted that in some alternative implementations, thefunctions/acts noted may occur out of the order noted in the figures.For example, two figures shown in succession may in fact be executedsubstantially concurrently or may sometimes be executed in the reverseorder, depending upon the functionality/acts involved.

Although the method operations were described in a specific order, itshould be understood that other operations may be performed in betweendescribed operations, described operations may be adjusted so that theyoccur at slightly different times or the described operations may bedistributed in a system which allows the occurrence of the processingoperations at various intervals associated with the processing.

Various units, circuits, or other components may be described or claimedas “configured to” or “configurable to” perform a task or tasks. In suchcontexts, the phrase “configured to” or “configurable to” is used toconnote structure by indicating that the units/circuits/componentsinclude structure (e.g., circuitry) that performs the task or tasksduring operation. As such, the unit/circuit/component can be said to beconfigured to perform the task, or configurable to perform the task,even when the specified unit/circuit/component is not currentlyoperational (e.g., is not on). The units/circuits/components used withthe “configured to” or “configurable to” language include hardware—forexample, circuits, memory storing program instructions executable toimplement the operation, etc. Reciting that a unit/circuit/component is“configured to” perform one or more tasks, or is “configurable to”perform one or more tasks, is expressly intended not to invoke 35 U.S.C.112, sixth paragraph, for that unit/circuit/component. Additionally,“configured to” or “configurable to” can include generic structure(e.g., generic circuitry) that is manipulated by software and/orfirmware (e.g., an FPGA or a general-purpose processor executingsoftware) to operate in manner that is capable of performing the task(s)at issue. “Configured to” may also include adapting a manufacturingprocess (e.g., a semiconductor fabrication facility) to fabricatedevices (e.g., integrated circuits) that are adapted to implement orperform one or more tasks. “Configurable to” is expressly intended notto apply to blank media, an unprogrammed processor or unprogrammedgeneric computer, or an unprogrammed programmable logic device,programmable gate array, or other unprogrammed device, unlessaccompanied by programmed media that confers the ability to theunprogrammed device to be configured to perform the disclosedfunction(s).

Any combination of one or more computer-usable or computer-readablemedia may be utilized. For example, a computer-readable medium mayinclude one or more of a portable computer diskette, a hard disk, arandom access memory (RAM) device, a read-only memory (ROM) device, anerasable programmable read-only memory (EPROM or Flash memory) device, aportable compact disc read-only memory (CDROM), an optical storagedevice, and a magnetic storage device. Computer program code forcarrying out operations of the present disclosure may be written in anycombination of one or more programming languages. Such code may becompiled from source code to computer-readable assembly language ormachine code suitable for the device or computer on which the code willbe executed.

Embodiments may also be implemented in cloud computing environments. Inthis description and the following claims, “cloud computing” may bedefined as a model for enabling ubiquitous, convenient, on-demandnetwork access to a shared pool of configurable computing resources(e.g., networks, servers, storage, applications, and services) that canbe rapidly provisioned (including via virtualization) and released withminimal management effort or service provider interaction and thenscaled accordingly. A cloud model can be composed of variouscharacteristics (e.g., on-demand self-service, broad network access,resource pooling, rapid elasticity, and measured service), servicemodels (e.g., Software as a Service (“SaaS”), Platform as a Service(“PaaS”), and Infrastructure as a Service (“IaaS”)), and deploymentmodels (e.g., private cloud, community cloud, public cloud, and hybridcloud).

The flow diagrams and block diagrams in the attached figures illustratethe architecture, functionality, and operation of possibleimplementations of systems, methods, and computer program productsaccording to various embodiments of the present disclosure. In thisregard, each block in the flow diagrams or block diagrams may representa module, segment, or portion of code, which comprises one or moreexecutable instructions for implementing the specified logicalfunction(s). It will also be noted that each block of the block diagramsor flow diagrams, and combinations of blocks in the block diagrams orflow diagrams, may be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions. These computerprogram instructions may also be stored in a computer-readable mediumthat can direct a computer or other programmable data processingapparatus to function in a particular manner, such that the instructionsstored in the computer-readable medium produce an article of manufactureincluding instruction means which implement the function/act specifiedin the flow diagram and/or block diagram block or blocks.

The foregoing description, for the purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the embodiments and its practical applications, to therebyenable others skilled in the art to best utilize the embodiments andvarious modifications as may be suited to the particular usecontemplated. Accordingly, the present embodiments are to be consideredas illustrative and not restrictive, and the invention is not to belimited to the details given herein, but may be modified within thescope and equivalents of the appended claims.

What is claimed is:
 1. A method comprising: building a bloom filter for each of a first set of files to be ingested into a data exchange to generate a set of bloom filters, wherein the data exchange includes a metadata storage where metadata including a list of files ingested is stored; storing the set of bloom filters in the metadata storage of the data exchange; and in response to receiving a set of candidate files to be ingested into the data exchange: generating file loading metadata for the set of candidate files; generating a reduced set of candidate files using minimum/maximum pruning based on the file loading metadata for the set of candidate files; and in response to generating the reduced set of candidate files, removing from within the reduced set of candidate files, by a processing device, each candidate file that is duplicative of a file in the first set of files using the set of bloom filters to generate a further reduced set of candidate files.
 2. The method of claim 1, wherein generating the reduced set of candidate files comprises: removing from the set of candidate files, one or more candidate files that are duplicative of a file in the first set of files using file loading metadata of each of the first set of files and the file loading metadata of the set of candidate files to generate the reduced set of candidate files.
 3. The method of claim 2, wherein the removing from within the reduced set of candidate files, each candidate file that is duplicative of a file in the first set of files comprises: for each candidate file of the reduced set of candidate files, processing the candidate file with each of the set of bloom filters to determine whether the candidate file is new or duplicative of any of the first set of files; and pruning each candidate file of the reduced set of candidate files that is determined to be duplicative.
 4. The method of claim 3, further comprising: grouping each candidate file of the reduced set of candidate files that is determined to be new together for ingestion into the data exchange.
 5. The method of claim 1, wherein storing the set of bloom filters in the metadata storage comprises: serializing the set of bloom filters; and storing the serialized set of bloom filters in a slice of the metadata storage dedicated to the set of bloom filters.
 6. The method of claim 1, further comprising: retrieving each of the set of bloom filters from the metadata storage; deserializing each of the set of bloom filters; and caching each of the set of bloom filters in a cache dedicated to the set of bloom filters.
 7. The method of claim 6, wherein each of the set of bloom filters is retrieved using a point lookup.
 8. The method of claim 1, wherein removing the one or more duplicative candidate files from the set of candidate files comprises: using a minimum and maximum pruning technique to remove duplicative candidate files from the set of candidate files based on the file loading metadata of each of the first set of files and the set of candidate files.
 9. A system comprising: a memory; and a processing device operatively coupled to the memory, the processing device to: build a bloom filter for each of a first set of files to be ingested into a data exchange to generate a set of bloom filters, wherein the data exchange includes a metadata storage where metadata including a list of files ingested is stored; store the set of bloom filters in the metadata storage of the data exchange; and in response to receiving a set of candidate files to be ingested into the data exchange: generate file loading metadata for the set of candidate files; generate a reduced set of candidate files using minimum/maximum pruning based on the file loading metadata for the set of candidate files; and in response to generating the reduced set of candidate files, remove from within the reduced set of candidate files, each candidate file that is duplicative of a file in the first set of files using the set of bloom filters to generate a further reduced set of candidate files.
 10. The system of claim 9, wherein to generate the reduced set of candidate files, the processing device is to: remove from the set of candidate files, one or more candidate files that are duplicative of a file in the first set of files using file loading metadata of each of the first set of files and the file loading metadata of the set of candidate files to generate a reduced set of candidate files.
 11. The system of claim 10, wherein to remove from within the reduced set of candidate files, each candidate file that is duplicative of a file in the first set of files, the processing device is to: for each candidate file of the reduced set of candidate files, process the candidate file with each of the set of bloom filters to determine whether the candidate file is new or potentially duplicative of any of the first set of files; and prune each candidate file of the reduced set of candidate files that is determined to be duplicative.
 12. The system of claim 11, wherein the processing device is further to: group each candidate file of the reduced set of candidate files that is determined to be new together for ingestion into the data exchange.
 13. The system of claim 9, wherein to store the set of bloom filters in the metadata storage, the processing device is to: serialize the set of bloom filters; and store the serialized set of bloom filters in a slice of the metadata storage dedicated to the set of bloom filters.
 14. The system of claim 9, wherein the processing device is further to: retrieve each of the set of bloom filters from the metadata storage; deserialize each of the set of bloom filters; and cache each of the set of bloom filters in a cache dedicated to the set of bloom filters.
 15. The system of claim 14, wherein the processing device retrieves each of the set of bloom filters using a point lookup.
 16. The system of claim 9, wherein to remove the one or more duplicative candidate files from the set of candidate files, the processing device is to: use a minimum and maximum pruning technique to remove duplicative candidate files from the set of candidate files based on the file loading metadata of each of the first set of files and the set of candidate files.
 17. A non-transitory computer-readable medium having instructions stored thereon which, when executed by a processing device, cause the processing device to: build a bloom filter for each of a first set of files to be ingested into a data exchange to generate a set of bloom filters, wherein the data exchange includes a metadata storage where metadata including a list of files ingested is stored; store the set of bloom filters in the metadata storage of the data exchange; and in response to receiving a set of candidate files to be ingested into the data exchange: generate file loading metadata for the set of candidate files; generate a reduced set of candidate files using minimum/maximum pruning based on the file loading metadata for the set of candidate files; and in response to generating the reduced set of candidate files, remove from within the reduced set of candidate files, by the processing device, each candidate file that is duplicative of a file in the first set of files using the set of bloom filters to generate a further reduced set of candidate files.
 18. The non-transitory computer-readable medium of claim 17, wherein to generate the reduced set of candidate files, the processing device is to: remove from the set of candidate files, one or more candidate files that are duplicative of a file in the first set of files using file loading metadata of each of the first set of files and the file loading metadata of the set of candidate files to generate a reduced set of candidate files.
 19. The non-transitory computer-readable medium of claim 18, wherein to remove from within the reduced set of candidate files, each candidate file that is duplicative of a file in the first set of files, the processing device is to: for each candidate file of the reduced set of candidate files, process the candidate file with each of the set of bloom filters to determine whether the candidate file is new or potentially duplicative of any of the first set of files; and prune each candidate file of the reduced set of candidate files that is determined to be duplicative.
 20. The non-transitory computer-readable medium of claim 19, wherein the processing device is further to: group each candidate file of the reduced set of candidate files that is determined to be new together for ingestion into the data exchange.
 21. The non-transitory computer-readable medium of claim 17, wherein to store the set of bloom filters in the metadata storage, the processing device is to: serialize the set of bloom filters; and store the serialized set of bloom filters in a slice of the metadata storage dedicated to the set of bloom filters.
 22. The non-transitory computer-readable medium of claim 17, wherein the processing device is further to: retrieve each of the set of bloom filters from the metadata storage; deserialize each of the set of bloom filters; and cache each of the set of bloom filters in a cache dedicated to the set of bloom filters.
 23. The non-transitory computer-readable medium of claim 22, wherein the processing device retrieves each of the set of bloom filters using a point lookup.
 24. The non-transitory computer-readable medium of claim 17, wherein to remove the one or more duplicative candidate files from the set of candidate files, the processing device is to: use a minimum and maximum pruning technique to remove duplicative candidate files from the set of candidate files based on the file loading metadata of each of the first set of files and the set of candidate files. 